PTL 1 discloses a sound source separation device that recovers source signals from a plurality of signals mixed in a space. The sound source separation device includes means for performing short-time Fourier transform on an observed signal, means for obtaining, through an independent component analysis, a separation matrix at each frequency at which short-time Fourier transform is performed, means for estimating an arrival direction of a signal taken from each row of the separation matrix at each frequency, means for determining whether its estimated value is fully reliable, and means for calculating a degree of similarity with respect to separation signals among the frequencies at which short-time Fourier transform is performed. Further included is means for, when resolving a permutation after a separation matrix is obtained at each frequency (replacement of a sound source at each frequency), determining the permutation by, at frequencies for which estimations of directions from which signals arrive are determined to be fully reliable, aligning the directions, and by, at other frequencies, increasing a degree of similarity with respect to separation signals at frequencies around the other frequencies. Therefore, while permutations are being resolved, source signals can be recovered.